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ABSTRACT 

Increasingly,  the  trend  for  military  and  other  orgardsations  is  an  increase  in  automation  to 
improve  the  efficiency  and  effectiveness  of  staff  in  the  workplace.  In  the  military  context, 
automated  tools  assisting  in  battlefield  visualisation  and  mission  planning  may  be 
implemented  for  these  very  purposes.  Automation  may,  however,  impact  on  the  performance 
of  military  teams  and  affect  the  situation  awareness  (SA)  necessary  for  good  team  functioning. 
This  review  examines  the  potential  impacts  and  benefits  of  automation  on  information 
sharing,  SA,  planning,  and  team  performance,  as  well  as  die  concept  of  measuring  SA.  Several 
potential  methods  for  measuring  both  shared  and  individual  SA  in  various  military  contexts 
(ie.  during  field  exercises.  Command  Post  Exercises  and  in  laboratory-based  experiments)  are 
then  briefly  outlined.  It  concludes  with  a  brief  discussion  of  the  need  for  further  examination 
of  the  impact  of  automation,  the  need  for  training  of  staff,  and  the  reality  of  the  benefits 
imparted  by  the  tools  themselves. 
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Executive  Summary 

The  current  trend  in  military  and  other  organisations  appears  to  be  an  increase  in  the 
use  of  automation  to  improve  the  efficiency  and  effectiveness  of  staff  in  the  workplace. 
In  the  military  context,  automated  tools  assisting  in  battlefield  visualisation  and 
mission  planning  may  be  implemented  for  these  very  purposes.  Automation  may, 
however,  impact  on  the  performance  of  military  teams  and  affect  the  situation 
awareness  (SA)  necessary  for  good  team  functioning. 

In  this  review,  the  potential  impacts  and  benefits  of  automation  on  information 
sharing,  SA,  planning,  and  team  performance  are  examined.  In  addition,  the  concept  of 
measuring  SA  -  which  has  been  problematic  in  terms  of  the  measurement  of  both 
individual  and  shared  SA  -  is  also  considered.  Several  potential  methods  for 
measuring  both  shared  and  individual  SA  in  various  military  contexts  (ie.  during  field 
exercises.  Command  Post  Exercises  and  in  laboratory-based  experiments)  are  then 
briefly  outlined,  along  with  their  potential  applications  in  the  various  military  contexts. 

The  review  then  concludes  with  a  brief  discussion  of  several  issues:  the  need  for 
further  examination  of  the  impact  of  automation,  the  need  for  training  of  staff,  and  the 
reality  of  the  benefits  imparted  by  the  tools  themselves.  The  information  contained  in 
the  review  should  form  a  foundation  for  considering  the  trialing  of  new  or  modified 
existing  SA  measurement  techniques,  which  should  lead  researchers  towards  the 
development  of  a  systematic  method  to  assess  the  contribution  of  automated  tools  to 
the  effectiveness  and  performance  of  teams. 
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1.  Automation:  the  way  ahead  for  military  decision 
makers  of  the  future? 


It  is  clear  that  there  is  an  increasing  emphasis  on  the  provision  of  automation  for  military 
command  and  control  (C2),  in  line  with  the  general  trend  towards  increasing  automation 
throughout  the  20'*'  century.  Automation  is  to  be  employed  in  many  of  the  operational 
aspects  of  the  military,  from  the  decision-making  processes  utilised  by  the  higher  ranks  to 
the  soldier  fighting  the  enemy  on  the  groimd.  What  is  often  at  issue  is  the  effectiveness  of 
such  automation:  are  these  technological  advancements  being  employed  effectively,  and 
are  they  facilitating  the  functioning  of  the  Army? 

In  some  cases,  the  answer  to  these  questions  may  well  be  'no'.  This  has  been  illustrated  by 
automation- facilitated  accidents  -  particularly  those  occurring  within  the  Naval  and  Air 
Force  elements  throughout  the  world  -  as  they  have  a  higher  reliance  on  automation  due 
to  their  very  nature.  The  USS  Vincennes  incident  illustrates  what  can  happen  when 
automation  exists,  and  yet  the  information  provided  via  this  means  is  either  ignored  by 
the  users  or  interpreted  in  a  biased  manner.  In  this  instance,  an  Iranian  airliner  carrying 
290  people  was  shot  down  by  the  crew  of  the  USS  Vincennes  in  the  belief  -  influenced  by 
the  fact  that  they  had  recently  been  under  attack  from  gunships  -  that  the  approaching 
aircraft  was  a  threat.  A  key  problem  in  situations  such  as  this  has  been  suggested  (Rochlin, 
1998)  to  be  the  difficulty  experienced  by  teams  in  maintaining  an  accurate  shared 
understanding  while  performing  with  high  technology  in  fast  moving  situations. 

In  recent  years  there  has  been  an  increased  emphasis  on  research  to  inform  the 
development  of  automated  tools,  particularly  from  the  Human  Factors  perspective.  This 
research  should  not  be  limited  to  the  design  of  such  systems  and  basic  measures  of  their 
usability.  The  need  for  training  in  the  effective  use  of  automated  tools  (to  facilitate  fast 
assimilation  and  efficient  performance  within  teams)  should  also  be  considered  in  the 
push  to  apply  automation  to  the  art  of  mission  planning  and  warfighting.  Additionally, 
there  have  been  large  bodies  of  work  generated  both  at  defence  research  organisations 
(such  as  DSTO  [Australia],  DCIEM  [Canada],  DERA  [UK])  and  at  external  organisations 
that  may  be  contracted  to  do  piecemeal  defence  work.  A  review  of  some  of  the  work  in 
this  vein  conducted  at  DSTO  in  recent  years  can  be  found  in  Seymour,  Grisogono, 
Unewisse,  Johnson,  Krieg  and  Haub  (2000). 

This  review  examines  a  series  of  related  issues,  including  the  use  of  information  for 
knowledge  generation,  the  generation  of  situation  awareness  (SA),  the  mission  plcinning 
process,  the  impact  of  automation  on  these  processes,  and  the  possibilities  of  enhanced 
planning  and  command  capabilities  with  the  implementation  of  automated  decision 
support  tools /systems.  Potential  evaluation  methods  for  SA  and  indicator  conditions 
(such  as  performance)  are  also  discussed.  This  is  in  keeping  with  the  aim  of  imderstanding 
the  impact  of  automation,  and  providing  a  means  through  which  to  form 
recommendations  to  ensure  that  this  impact  is  a  beneficial  one. 
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2.  Knowledge  and  information  use 

It  is  well  known  that  knowledge  and  information! — two  distinctly  different  entities — are 
critical  factors  in  the  planning  of  missions  and  indeed  in  any  decisions  made  by 
commanders  with  respect  to  the  deployment  of  troops  and  equipment.  It  is  also  apparent 
that  engineers  designing  automated  decision  support  tools  and/or  systems  need  to 
consider  the  warfighter's  point  of  view,  particularly  in  terms  of  asking  how  such  tools  will 
assist  the  commander/ plaiming  team/fighter  in  reaching  their  goals  and  objectives  more 
quickly  and  effectively,  and  with  less  binding  of  physical  and  cognitive  resources  than  the 
current  paper  and  pencil  method.  Too  often,  the  implementation  of  automation  has  the 
potential  to  result  in  excessive  or  extraneous  information  being  presented  to  the 
commander  or  planning  team,  which  can  increase  the  pressure  experienced  imder  time- 
constrained  working  conditions.  Although  experienced  decision-makers  tend  to  filter  the 
necessary  information  and  therefore  do  not  suffer  as  much  as  inexperienced  decision¬ 
makers,  there  still  exists  the  potential  for  even  these  expert  planners  and  commanders  to 
be  swamped  by  the  excessive  information  provided  by  new  technologies.  As  implied  in  a 
paper  by  Seymour  et  al  (2000),  a  digital  tool  (for  example,  a  computer  display  containing  a 
map  and  all  the  relevant  entities  on  the  battlefield)  can  lead  to  information  overload  due  to 
" . .  .too  much  detail  swamping  a  user  and  leading  to  poorer  and  slower  decision  making." 
This  phenomenon  has  been  noted  by  researchers  under  various  conditions  (including 
during  microworld  simulations  such  as  'FireChief')  and  is  outlined  in  Omodei,  McLennan 
and  Wearing  (2000)  and  Rooney,  Kallmeier  and  Stevens  (1998).  These  results  do  not, 
however,  indicate  that  all  'digitisation'2  is  detrimental  to  operational  effectiveness.  It  is 
simply  that  the  application  of  technology  to  such  processes  as  these  requires  care  in  terms 
of  accommodating  human  capabilities  and  limitations. 

When  looking  at  the  information  needs  of  commanders,  Kahan,  Worley  and  Stasz  (1989) 
indicate  that  commanders  seek  a  dynamic  image  of  the  battlefield  that  will  lead  to  an 
understanding  of  what  actions  are  necessary.  Depending  on  the  situation,  this  image  tends 
to  be  composed  of  between  five  and  nine  major  components,  including  the  traditional 
factors  of  METT-T  (Mission,  Enemy,  Terrain,  [own]  Troops,  and  Time  available).  For  the 
effective  implementation  of  command,  however,  a  commander's  staff  must  —  to  some 
extent  —  also  hold  the  mental  image  held  by  the  commander:  in  this  way,  his  staff  can 
supply  relevant  and  important  incoming  information  in  a  timely  and  effective  manner. 
This  then  necessitates  a  focus  on  the  communication  of  information  and/ or  knowledge 
within  the  C2  process,  so  that  the  communication  of  mental  images  can  be  facilitated. 


!  That  is,  'information'  represents  the  facts  or  data  collected,  while  'knowledge'  represents  the  sum 
or  range  of  what  has  been  perceived/discovered/leamed.  Knowledge  therefore  represents  the 
persons'  understanding  of  information  they  have  obtained  or  been  presented  with,  and  implies  that 
some  type  of  fusion /integration/processing  of  raw  information  has  taken  place. 

2  'Digitisation'  is  the  application  of  information  and  commtmication  technology  in  the  battlefield 
which  allows  data  to  be  entered  directly  into  a  system  by  sources,  and  automatically  disseminated 
to  users 
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Information  is  obtained  from  multiple  sources  throughout  the  course  of  mission  planning 
and  execution,  though  the  rate  of  information  influx  tends  to  vary  during  the  fighting 
phase  of  battle  activities.  Enemy  locations  and  activities  are  reported  as  they  come  to  hand, 
as  are  the  status  of  friendly  units,  but  there  tends  not  to  be  a  continuous  flow  of  such 
information  because  battle  is  not  a  smooth,  continuous  flow  of  fighting.  This  new 
information  must  be  integrated  into  the  picture  the  commander  holds  of  the  battle  itself, 
and  then  applied  to  adjust  courses  of  action  when  necessary.  This  integration  expands  the 
commander's  knowledge  of  the  situation  and  generally  facilitates  his  ability  to  predict 
necessary  courses  of  action. 


3.  Situation  Awareness 


Situation  awareness  is  a  term  used  originally  in  the  aircraft  community,  and  achieving  it  is 
perhaps  the  most  difficult  aspect  of  an  operator's  work.  It  has  been  defined  by  Endsley 
(1988:  97)  as:  "...the  perception  of  the  elements  within  a  volume  of  time  and  space,  the 
comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future."  It  is 
therefore  clear  that  SA  is  necessary  in  order  for  people  to  perform  tasks  effectively 
(Endsley,  2000).  Dominguez  (1994)  gives  a  similar  definition,  but  places  more  emphasis  on 
the  impact  of  awareness  on  cue  extraction  and  directed  perception  (i.e.  its  contribution  to 
attention).  For  Dominguez  (1994:  11),  then,  SA  constitutes  a  "continuous  extraction  of 
environmental  information,  integration  of  this  knowledge  to  form  a  coherent  mental 
picture,  and  the  use  of  that  mental  picture  in  directing  further  perception  and  anticipating 
future  events."  Thus,  SA  appears  to  involve  three  hierarchical  levels  linked  by  various 
cognitive  processes.  These  levels  are: 

•  Perceiving  critical  factors  in  the  environment  [level  1], 

•  Understanding  what  those  factors  mean  (in  particular  when  integrated  together  in 
relation  to  the  person's  goals)  [level  2],  and 

•  An  understanding  of  what  will  happen  with  the  system  in  the  near  future  [level  3]. 

The  higher  levels  of  SA  allow  operators  to  function  in  a  timely  and  effective  manner  when 
the  demands  of  high  workload  periods  prohibit  effective  use  of  explicit  coordination 
methods  (Endsley,  1996).  Level  3  SA  cannot  be  developed  or  employed,  however,  unless 
levels  1  and  2  are  already  well  established  in  the  individual. 

This  definition  of  SA  refers  to  individual  SA:  it  is  not  enough  for  successful  mission 
planning  or  execution,  however,  that  SA  exists  individually  in  planning  team  members.  It 
must  be  shared  SA  in  order  to  facilitate  effective  planning  based  on  a  common 
imderstanding  of  both  the  situation  and  the  implications  of  any  actions /orders  for  action. 
In  addition,  it  is  not  adequate  to  simply  share  a  few  common  elements  of  SA  between  team 
members  or  across  teams.  Shared  SA  should  be  a  relatively  complete  and  integrated 
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picture  of  the  current  situation  and  the  potential  developments  to  that  situation,  as  well  as 
the  possible  tactics  to  be  used  in  the  event  that  those  developments  do  occur. 


3.1  Factors  in  shared  SA 

Situation  awareness  in  aircrew  members  from  a  variety  of  aviation  commimities  (i.e. 
military,  general  and  carriers)  was  examined  by  Prince  and  Salas  (1998)  via  a  series  of 
interviews.  The  outcomes  of  these  interviews  indicated  that  there  were  four  main  factors 
contributing  to  the  development  and  maintenance  of  SA  in  these  crews,  and  that  these 
varied  in  importance  according  to  the  level  of  experience  of  the  pilots.  Those  with  less  than 
1000  hours  flying  time  considered  that flight  preparation  {planning)  and  communication  were 
the  two  most  important  factors  for  SA,  while  pilots  with  more  than  1000  hours  tended  to 
emphasise  leadership  and  adaptability  behaviours  were  more  important  for  team/ shared  SA 
(although  acknowledging  that  communication  and  preparation  were  also  of  importance). 

Thus,  these  interviews  revealed  that  aircrew  members  perceived  team  or  shared  SA  to  be 
achieved  and  maintained  through  a  combination  of  individual  SA  and  the  four  factors 
(termed  'process  skills')  outlined  above  (Prince  and  Salas,  1999).  This  has  been  confirmed 
by  research  findings  indicating  that  crew  members'  SA  is  strongly  related  to  the  team 
process  skills  identified  by  the  interviews  (Brannick,  Ellis,  Prince  and  Salas,  in 
preparation). 

The  concept  of  shared  mental  models  is  a  construct  that  can  help  to  provide  an 
understanding  of  the  relationship  between  team  processes  and  shared  situational 
awareness  in  teams  (Salas,  Stout  and  Cannon-Bowers,  1994).  In  terms  of  what  aspects  of 
mental  models  may  have  to  be  shared  in  order  to  facilitate  the  team  process,  researchers 
have  proposed  that  successful  teams  share  knowledge  of  several  common  factors  such  as 
overall  task  and  team  goals,  individual  tasks,  team  member  roles,  and  the  team  members 
themselves  (Cannon-Bowers,  Salas  and  Converse,  1995).  Developing  this  proposal. 
Converse  and  Kahler  (1992)  suggest  -  based  on  an  extensive  literature  review  -  that  there 
are  three  types  of  knowledge  that  can  be  shared  by  team  members,  and  that  these  are 
declarative,  procedural  and  strategic.  They  defined  these  three  types  in  the  following  way: 

•  Declarative:  these  models  contain  information  about  the  concepts  and  elements  in  the 
domain  and  the  relationships  between  them.  They  include  knowledge  of  facts,  rules 
relationships  and  the  common  factors  listed  by  Cannon-Bowers  et  al  (1995)  previously. 

•  Procedural:  these  models  store  information  about  the  steps  to  be  taken  (and  the  order  m 
which  to  take  them)  in  order  to  accomplish  various  activities.  It  is  essentially  timing 
and  sequence  information. 

•  Strategic:  these  models  consist  of  information  that  is  the  basis  of  problem  solving,  like 
actions  plans  to  achieve  goals,  knowledge  of  the  context  in  which  to  implement  the 
procedures,  actions  upon  failure  of  the  proposed  solution,  and  responses  to  absent 
information.  This  is  a  compilation  of  declarative  and  procedural  knowledge,  and 
through  team  experience  this  knowledge  allows  for  automatic  performance  (and 
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expert  team  performance)  through  the  use  of  the  correct  task  strategies  (Converse  and 
Kahler,1992:  pp.5-6). 

In  this  scheme  then,  the  strategic  model  applies  knowledge  stored  in  the  procedural  and 
declarative  models  within  a  dynamic  mission  context  to  allow  an  understanding  of  the 
necessary  actions,  cues  and  sequences  needed  to  achieve  the  team  and  task  goals  (Cannon- 
Bowers  et  al.  1995). 

In  summarising  the  relevant  SA-related  literature,  Cannon-Bowers  et  al.  (1995)  make  the 
following  important  points  about  the  nature  of  SA: 

1.  Mental  models  are  important  for  individual  SA. 

2.  Shared  mental  models  are  important  to  team  performance  because  they  allow  team 
members  to  form  the  necessary  explanations  and  expectations  of  team  and  task 
actions,  and 

3.  Team  (shared)  SA  is  important  to  team  performance. 

Clearly,  then,  a  shared  mental  model  is  an  important  prerequisite  to  attaining  shared  SA 
(Sarter  and  Woods,  1991).  Simply  having  a  shared  mental  model  is  not  a  guarantee  that 
adequate  shared  SA  will  be  developed,  however.  There  are  other  issues  (like  shared 
cognition,  for  example)  to  be  considered  before  the  process  of  transforming  shared 
information  into  shared  SA  can  be  understood. 


3.2  Challenges  to  shared  SA 

Shared  SA  is  not  simple  to  develop,  and  can  be  adversely  affected  by  a  range  of  factors, 
including  stress,  mental  work  over-  or  imder-load,  system  design,  complexity,  human 
error  in  perception,  and  automation.  For  example,  it  is  well  documented  in  psychological 
literature  that  a  certain  level  of  stress  can  enhance  performance,  but  that  excessive  or 
prolonged  higher  stress  levels  can  have  the  opposite  effect  due  to  demands  on  attentional 
processes  and  the  physical  side  effects  (Hockey,  1986).  Thus,  a  lower  attention  capacity 
and  narrowing  of  attention  spans  (leading  to  concentration  on  a  limited  number  of 
factors),  disruptions  of  scan  patterns,  and  premature  closure  (making  a  premature  decision 
without  considering  all  available  information)  can  be  problematic,  particularly  for  level  1 
SA  (Endsley,  1999).  In  addition,  a  decrease  in  working  memory  capacity  and  hindrance  of 
information  retrieval  can  also  adversely  affect  SA  (Hockey,  1986). 

Errors  in  both  individual  and  shared  SA  can  be  attributed  to  various  factors  or  behaviours, 
not  the  least  of  which  could  be  the  introduction  of  automation.  Endsley  (1995)  has  created 
a  taxonomy  of  SA  errors  (see  also  Endsley,  1999).  These  are  categorised  into  the  three 
levels  of  SA,  and  the  percentages  of  SA  error  accounted  for  by  the  various  behaviours  is 
displayed  in  tabular  form.  The  three  categories  are  as  follows: 
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SA  Error  Type 

Level  1:  Failure  to  correctly  perceive  the  situation 

Level  2:  Failure  to  comprehend  the  situation 

Level  3:  Failure  to  project  the  situation  into  the  future 

By  far  the  highest  percentage  of  SA  error  overall  is  accounted  for  by  a  failure  to  monitor  or 
observe  data  (which  is  in  the  level  1  SA  category);  in  the  level  2  SA  category,  the  highest 
percentage  of  error  is  accoimted  for  by  use  of  an  incorrect  mental  model.  In  the  level  3  SA 
category,  it  appears  to  be  poor  or  over-projection  of  the  current  trends  leads  to  the 
majority  of  SA  errors.  Care  must  be  taken,  however,  to  ensure  that  these  problems  with  SA 
are  not  simply  labelled  'human  error',  and  that  researchers  differentiate  machine-related 
and  human-related  causes  for  SA  error  (Baxter  and  Bass,  1998). 

It  is  conjectured  that  automation  can  adversely  affect  SA  due  to  the  tendency  for 
automated  tools  to  put  the  human  'out  of  the  loop'.  Automation  could  result  in  pilots 
being  slower  to  detect  and  respond  to  problems,  for  example;  however  automation  may 
not  have  a  consistently  adverse  effect  across  the  board.  It  may  prove  beneficial  in 
situations  where  (a)  automation  enhances  information  or  data  fusion  with  no  additional 
workload  for  the  system  operator,  or  (b)  where  either/both  mental  or  manual  workload 
can  be  reduced.  It  is  these  types  of  benefits  that  are  sought  for  the  military  community  in 
particular,  as  any  improvements  in  the  speed  of  information  processing  may  have  flow-on 
effects  leading  to  shorter  decision  cycles.  These  may  therefore  lead  to  decision  and 
planning  operations  (followed  by  the  execution  of  missions)  being  conducted  within  the 
decision  cycle  of  the  enemy  force  by  friendly  forces.  Obviously  a  time  saving  at  this 
preliminary  stage  should  provide  advantages  for  friendly  forces  in  terms  of  their  ability  to 
conduct  successful  missions. 

Another  challenge  to  the  development  and  maintenance  of  SA  in  teams  is  lack  of 
appropriate  communications  between  members.  Schwartz  (1990)  asserts  (in  terms  of 
aircrew)  that  although  each  member  in  the  cockpit  has  individual  SA,  the  group's  shared 
SA  can  be  limited  by  the  working  memory  of  the  individuals  and  so  must  be  constructed 
and  updated  by  information  updates  from  each  of  the  team  members  in  order  to  maintain 
an  appropriate  level  of  SA.  In  this  way,  the  crew's  SA  is  believed  to  be  a  direct 
consequence  of  (among  other  things)  the  communication  ability  of  the  crew.  This  is 
supported  by  findings  from  Orasanu  (1990)  in  a  study  of  crew  decision-making.  She  found 
that  crews  that  performed  better  on  the  simulated  mission  made  a  significantly  greater 
number  of  SA  statements  than  did  crews  that  performed  poorly.  Mosier  and  Chidester 
(1991)  studied  communications  during  emergency  situations,  proposing  that  the  effects  of 
differences  in  communications  would  be  more  apparent  under  these  conditions.  They 
foimd  that  the  number  and  type  of  communications  did  in  fact  relate  to  crew  performance, 
and  that  crews  collected  information  before  as  well  as  after  the  decisions  were  made, 
indicating  that  although  the  problem  might  have  been  solved,  it  was  not  forgotten.  Further 
research  to  clarify  the  role  of  communication  in  SA  development  and  maintenance  needs 
to  be  conducted,  as  the  research  to  date  has  not  conclusively  elucidated  the  role  it  plays  in 
this  vital  component  of  teams. 
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3.3  The  concept  of  measuring  SA 

It  is  not  adequate  to  discuss  SA,  shared  SA  and  the  effects  of  various  factors  on  these 
constructs  without  a  discussion  of  the  means  available  for  measuring  the  construct  itself. 
This  is  where  the  main  difficulty  of  research  in  SA  is  found:  individual  SA  is  a  relatively 
difficult  construct  to  measure,  and  thus  shared  or  team  SA  presents  somewhat  more  of  a 
problem  in  terms  of  the  added  complexity  of  group  dynamics. 

An  additional  difficulty  is  quantifying  how  much  SA  is  "enough".  Endsley  (2000)  states 
that  SA  and  performance  measures  are  only  linked  probabilistically,  and  that  there  are  no 
set  thresholds  of  SA  to  guarantee  a  given  level  of  performance.  Other  factors  may  assist  in 
determining  outcomes,  and  so  a  team  with  a  low  level  of  SA  may  perform  well  due  to  a 
high  degree  of  luck.  She  proposes  that  an  answer  to  this  is  to  be  concerned  with  relative 
levels  of  SA.  In  terms  of  the  automation  issue,  this  would  mean  an  increase  in  SA  (after  the 
addition  of  new  technology)  relative  to  the  level  measured  prior  to  this  introduction. 
Measurement  should  also  be  against  the  "ideal"  SA:  that  is,  perfect  knowledge  on  all 
relevant  aspects  of  a  situation,  with  no  gaps  or  holes  in  it.  This  means  that  SA  spans  a 
broad  set  of  constructs  and  is  not  a  simple  concept,  which  makes  the  job  of  measurement 
and  enhancement  of  SA  a  complex  one.  In  addition,  actual  SA  tends  to  be  a  subset  of  the 
achievable  SA,  which  is  again  a  subset  of  the  ideal  SA  of  any  situation  (Pew,  2000):  given 
the  complexities  of  the  world  (and  particularly  the  military  battle  situation),  perfect  SA  is 
generally  not  feasible,  and  operators  would  be  in  the  higher  percentages  of  achievement  if 
they  did  indeed  achieve  the  entirety  of  theoretically  achievable  SA. 

When  dealing  with  individual  SA,  Entin  (1998)  states  that  an  individual's  level  of  SA  will 
be  partially  determined  by  the  quantity  and  quality  of  the  information  available  and  by  the 
individual's  ability  to  utilise  the  important  information  sources  and  fill  in  for  missing, 
imprecise  or  incomplete  information.  Entin  (1998)  and  Entin,  Serfaty  and  Entin  (1995)  list 
reasons  for  lowered  SA  as  encompassing  factors  such  as  lack  of  anticipation  of  —  or 
attention  to — vital  factors  (due  to  lack  of  awareness  of  the  importance  of  these  factors,  or 
attention  being  focused  on  other  things) .  Entin  (1998)  gives  two  t5q?es  of  SA  measure:  high- 
level  and  detailed,  both  of  which  are  taken  at  multiple  points  in  a  simulated  attack.  High- 
level  measures  assess  SA  according  to  subjects'  responses  to  general  questions  about  the 
evolving  tactical  situation.  The  detailed  SA  measure  is  based  on  questions  about  elements 
of  the  situation  (in  a  way  similar  to  measures  derived  from  the  Situation  Awareness  Global 
Assessment  Techiuque  [SAGAT],  which  is  based  on  a  broad-ranging  assessment  of 
operator  SA  requirements)  (Strater  and  Endsley,  2000). 

Measurements  by  Entin  (1998)  suggest  that  detailed  SA  is  more  integrated  and  less 
ambiguous  during  the  planning  phase,  as  the  situation  is  more  static  and  time  is  not 
perceived  to  be  a  critical  factor.  At  this  point,  there  is  more  time  available  to  fine  time  the 
details  of  SA  and  resolve  any  ambiguity  or  imcertainty  present.  As  the  mission  proceeds, 
however,  the  situation  becomes  more  fluid  and  events  occur  more  rapidly.  Individuals 
may  not  have  time  to  integrate  all  the  incoming  information  into  a  coherent  picture,  so  the 
level  of  detailed  SA  is  less  consistent  at  later  points  in  the  mission.  High  level  SA,  by 
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contrast,  did  not  show  this  pattern  and  supported  the  notion  of  strong  relationships 
between  levels  of  SA  in  adjacent  phases  (i.e.  the  planning,  ingress  and  battle  position 
phases)  (Entin,  1998:  251  -  252).  Entin  suggests  that  this  may  indicate  a  difference  in  the 
cognitive  processes  in  the  two  types  of  SA.  That  is,  high  level  SA  may  depend  more  on  the 
accuracy  of  the  individuals  mental  model  of  the  situation,  while  detailed  SA  may  depend 
more  on  the  individual's  perceptual  skills,  memory  and  ability  to  extract  and  integrate 
specific  pieces  of  information. 

3.4  Possible  measures  of  SA 

Two  types  of  SA  measure  emerged  from  Entin's  (1998)  study  of  detailed  and  high  level 
SA.  Measures  of  detailed  SA  were  based  on  the  responses  of  subjects  to  questionnaires  that 
included  questions  designed  to  capture  specific  elements  of  the  situation.  These  elements 
included  current  and  projected  locations  of  specific  friendly  and  enemy  units,  status  and 
relative  location  of  ownership,  and  the  expected  direction  of  enemy  movement.  Questions 
about  location  or  direction  of  movement  were  answered  on  the  tactical  simulation  display, 
with  other  questions  being  answered  in  writing.  Subject  matter  experts  (SMEs)  were  used 
to  determine  the  precision  required  for  a  correct  response  (Entin  (1998)  p.  251).  The 
subjects'  answers  were  evaluated  on  a  four-point  scale  in  terms  of  correctness,  and  a 
weighted  average  of  the  scores  on  individual  questions  was  computed  to  give  an  overall 
measure  of  detailed  SA. 

Measures  of  high-level  SA  were  gathered  via  a  verbally  administered  questionnaire 
containing  between  five  and  eight  open-ended  questions  about  aspects  of  the  tactical  plan, 
constraints  imposed  by  geographic  situation,  implications  of  enemy  actions,  and 
indications  of  the  plan's  success.  The  high-level  SA  measure  was  derived  by  subject  matter 
expert  (SME)  evaluation  (on  a  four-point  scale)  of  subjects'  answers  about  the  current  and 
evolving  tactical  situation  on  the  five  components  of  SA  (i.e.  accuracy  of  friendly  and 
enemy  dispositions,  assessment  of  changes  to  friendly  and  enemy  situations  over  time, 
and  grasp  of  the  evolving  tactical  plan).  The  overall  score  was  obtained  by  taking  the 
arithmetic  mean  of  the  five  component  scores. 

There  are  several  means  by  which  SA  can  be  measured,  according  to  Pew  (2000: 38  -  43), 
which  fall  into  the  following  categories. 

Direct  systems  performance  measures.  These  are  only  applicable  in  very  limited  situations. 
The  use  of  scenarios  specifically  designed  to  create  openings  for  unobtrusive  performance 
measures  to  be  taken  on  specific  SA  can,  however,  be  effectively  substituted  for  these.  Use 
of  such  scenarios  may  involve  the  introduction  of  imexpected  events,  and  measuring  the 
time  taken  for  the  subject  to  react,  as  well  as  the  quality  of  the  reactive  measures  taken. 
Alternatively,  disruptions  intended  to  disorient  the  team  or  operator  can  be  introduced. 
The  team's  or  operator's  recovery  time  and  the  success  of  recovery  tactics  taken  can  be 
measured. 
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Direct  experimental  techniques.  This  is  the  most  commonly  used  measurement  method,  and 
involves  using  queries  and/ or  probes  along  with  measures  of  information  seeking  during 
ongoing  task  performance.  This  does,  however,  require  that  the  pace  be  relatively  slow  or 
that  frequent  periods  of  inactivity  are  available.  Another  common  method  of  performing 
these  direct  experimental  techniques  involves  freezing  the  scenario/ simulation  and  asking 
the  necessary  questions  before  beginning  the  scenario  once  more  (as  in  the  SAGAT 
technique  formalised  by  Endsley,  1988).  Problems  here  include  the  disruption  caused  by 
the  freeze  technique  and  the  possible  alteration  in  behaviour  due  to  subjects  expecting  the 
probes  (e.g.  preparation  for  future  and  increased  focus  on  potential  probe  issues).  3 

Verbal  protocols.  This  method  involves  the  direct  recording  of  information  from  observers 
or  subjects  during  or  immediately  after  an  exercise,  or  from  videotape  records  of  the 
exercise.  Here,  subjects  may  be  asked  to  think  aloud  or  explain  information  as  they  work. 
This  method  is  most  valuable  at  the  beginning  of  an  evaluation  when  concepts  are  still 
being  refined. 

Subjective  measures.  These  methods  include  self-assessments,  expert  judgements,  peer 
ratings,  and  supervisor  or  instructor  ratings.  The  best  of  these  techniques  to  date  appears 
to  be  the  self-administered  SA  test,  the  SART  (Situation  Awareness  Rating  Technique), 
involving  a  three  scale  subset  the  results  of  which  are  combined  via  algebraic  equation  to 
produce  an  overall  estimate  of  subject  SA  (see  Taylor,  1990). 

In  addition,  the  SALIANT  methodology  developed  by  Muniz,  Stout,  Bowers  and  Salas 
(1998)  and  validated  by  Bowers,  Weaver,  Barnett  and  Stout  (1998)  has  also  been  put 
forward  as  a  successful  technique  for  producing  checklists  for  behaviourally  assessing  SA. 
The  SALIANT  acronym  refers  to  Situational  Awareness  Linked  Indicators  Adapted  to 
Novel  Tasks,  and  involves  the  use  of  a  theoretically  based  list  of  behaviours  (known  to  be 
associated  with  SA)  to  assess  team  behaviour.  A  set  of  team  behaviours  theorised  by 
Mxmiz  et  al  (1998)  to  be  linked  with  team  SA  is  included  in  Table  1  below. 

As  Table  1  shows,  there  are  many  observable  behaviours  that  can  potentially  be  used  as 
indicators  of  team  or  shared  SA.  These  are  not  complete  in  their  inferential  validity, 
however,  and  should  be  supplemented  by  some  form  of  direct  questioning  of  subjects, 
which  may  include  a  "video  walkthrough"  technique  if  circumstances  allow.  This  is 
particularly  true  in  the  case  of  shared  SA  during  the  battle  itself.  It  may  be  that,  during 
planning  sessions,  the  unobtrusive  methods  yield  adequate  information  for  the  observers 
/  researchers  to  enable  SA  measurement. 


3  Please  refer  to  French  and  Hutchinson  (2002)  for  detail  on  the  modified  SAGAT  methodology 
applied  to  the  measurement  of  SA. 
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Table  1.  Behavioural  indicators  of  team  situation  awareness 

Demonstrated  awareness  of  surrounding  environment _ 

>  Monitored  environment  for  changes,  trends,  abnormal  conditions 

>  Demonstrated  awareness  of  where  he/she  was _ 

Recognised  problems _ 

>  Reported  problems 

>  Located  potential  sources  of  problem 

>  Demonstrated  knowledge  of  problem  consequences 

>  Resolved  discrepancies 

>  Noted  deviations _ 

Anticipated  a  need  for  action _ _ 

>  Recognised  a  need  for  action 

>  Anticipated  consequences  of  actions  and  decisions 

>  Informed  others  of  actions  taken 

>  Monitored  actions _ _ 

Demonstrated  knowledge  of  tasks _ _ 

>  Demonstrated  knowledge  of  tasks 

>  Exhibited  skilled  time  sharing  attention  among  tasks 

>  Monitored  workload 

y  Shared  workload  within  station 

>  Answered  questions  promptly _ 

Demonstrated  awareness  of  information _ 

>  Commimicated  important  information 

>  Confirmed  information  where  possible 

>  Challenged  information  when  doubtful 

>  Re-checked  old  information 

>  Provided  information  in  advance 

>  Obtained  information  on  what  was  currently  happening 

>  Demonstrated  understanding  of  complex  relationships 

>  Briefed  status  frequently _ _ 


This  would  be  preferable  to  applying  more  direct  questioning  methods,  as  the  less 
obtrusive  the  data  gathering  technique,  the  higher  the  quality  of  the  data.  Minimising  the 
subjects'  contact  with  the  concepts  researchers  are  trying  to  measure,  and  attempting  to 
create  the  least  possible  disruption  to  the  team  processes  are  desirable  here.  More 
obtrusive  measures  may  produce  measures  of  what  the  subjects  think  the  researchers  want 
to  see  or  hear,  rather  than  a  picture  of  what  would  actually  have  been  the  case  without 
researcher  interference.  This  type  of  problem  has  been  combated  by  researchers  such  as 
Endsley  (2000)  (for  example)  via  the  use  of  randomised  and  comprehensive  questions 
during  the  use  of  the  freeze  technique.  In  order  to  explore  the  measurements  possible 
during  planning,  the  concepts  involved  in  planning  will  now  be  explored  in  more  detail. 
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4.  Mission  planning 

The  architecture  of  C2  operations  is  important  to  the  imderstanding  of  which  functions 
will  be  important  in  the  mission  planning  process,  and  which  of  these  may  be  effectively 
enhanced  by  the  addition  of  automated  tools  to  the  process.  Noble  (2000)  has  represented 
this  architecture  diagrammatically,  as  shown  in  Figure  1.  It  is  apparent  in  this  figure  that 
one  of  the  keys  to  effective  operational  architecture  is  the  manner  in  which  it  integrates 
situation  assessment,  planning  and  execution.  In  this  diagram,  "understanding"  refers  to 
the  understanding  of  how  the  situation  impacts  the  mission,  "decision"  is  embodied  in  the 
commander's  directive  and  includes  the  clear  expression  of  intent,  and  "feedback"  refers 
to  the  information  needed  to  determine  how  well  the  plan  is  progressing  and  to  identify 
opportunities  and  risks  (Noble,  2000). 


Figure  1:  Elements  ofC2  operational  architecture,  emphasising  richly  coupled,  dynamic  interfaces 
(adapted  from  Figure  2  in  Noble,  2000) 


On  examination  of  the  'lessons  learned'  data  from  the  Marine  Corps  MAGTF  Staff 
Training  Program  (1999),  it  was  discovered  that  most  of  the  identified  operational 
shortfalls  existed  due  to  human  performance  limitations.  Thus,  four  groups  of 
recommendations  relating  to  these  shortfalls  were  put  forward  as  follows: 

a.  Building  and  sharing  a  common  picture 

b.  Developing  mission  strategy/ concept  and  identifying  needed  tasks 

c.  Establishing  conditions  needed  for  adaptive  control 

d.  Expressing  intent  in  a  clear  directive  (Noble,  2000: 10) 
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Clearly,  item  (a)  in  the  list  above —building  and  sharing  a  common  picture  —  is  essential 
to  shared  SA.  It  emerges  from  the  team's  performance  of  the  situation  assessment 
functions  that  are  a  prelude  to  the  planning  phase,  and  is  susceptible  to  the  types  of 
human  errors  inherent  in  all  conceptual  entities  that  rely  on  multiple/ shared  human 
perceptions.  Differing  interpretations  of  a  situation  (or  an  aspect  of  a  situation)  or  the 
actions  necessary  to  deal  with  that  situation  may  lead  to  quite  different  individual  SA,  or 
team  SA  if  referring  to  intra-team  shared  SA. 

4.1  Situation  assessment  and  planning  functions 

Noble  (2000)  also  outlines  a  series  of  mid-level  situation  assessment  and  planning 
functions  that  are  divided  and  organised  under  higher  level  function  headings  as  set  out  in 
Table  2.  It  can  be  observed  here  that  the  situation  assessment  functions  lead  directly  into 
the  plarming  functions  (and  outcomes).  That  is,  the  SA  that  must  be  maintained  by  the 
mission  planners  should  be  as  complete  as  possible  to  facilitate  effective  planning 
outcomes.  This  relies  on  the  adequacy  of  the  situation  assessment,  particularly  in  terms  of 
the  information  being  entered  into  the  picture,  and  the  knowledge  this  information  then 
generates.  In  addition,  the  use  of  a  clear  intent  from  the  higher  commander  is  vital  to  the 
production  of  a  mission  plan,  and  thus  the  process  relies  heavily  on  the  integration  of  this 
information  with  the  SA-producing  information  coming  in  from  other  sources. 

On  examining  the  planning  functions,  it  is  evident  that  the  plarming  team  would  rely  on  a 
well-developed  and  maintained  level  of  shared  SA  in  order  to  carry  out  many  of  these 
plarming  functions.  This  is  particularly  true  in  the  case  of  coordination  with  other 
distributed  team  members  or  teams,  and  can  be  difficult  to  maintain  under  these 
circumstances.  This  is  one  of  the  conditions  xmder  which  a  visualisation  tool,  which 
provides  a  fast  updating  visual  image^  to  disparate  teams,  can  be  most  effective  in 
enhancing  SA. 

Planning  functions  do,  however,  enjoy  a  positive  relationship  with  the  development  and 
maintenance  of  shared  mental  models  in  team  members,  according  to  Stout,  Caimon- 
Bowers,  Salas  and  Milanovich  (1999).  These  mental  models  in  turn  contribute  to  both  the 
development  and  maintenance  of  shared  SA  in  teams.  As  such,  mission  planning 
promotes  the  sharing  of  information,  priorities  and  goals  related  to  the  task  requirements, 
and  team  member's  (or  team's)  roles  can  be  clarified  prior  to  mission  commencement. 
Stout  et  al  (1999)  found  that  effective  plarming  prior  to  mission  execution  enhanced  the 
effectiveness  of  teams  (see  also  Sterling  and  Lickteig,  2000). 


*  An  image  that  combines  text,  symbols,  and  maps,  for  example. 
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Table  2:  The  mid-level  C2  functions  (modified  from  Table  1  in  Noble,  2000) 

Situation  Assessment  Functions 

S-1.  Understand  situation  assessment  requirements. 

a)  Identify  elements  of  requested  information 

b)  Review  context  of  requested  information 

S-2.  Gather  the  information 

a)  Identify  the  information  to  be  gathered 

b)  Formulate  and  make  requests 

c)  Assess  information  quality 

d)  Screen,  organise,  and  forward  information 

S-3.  Estimate  the  situation 

a)  Assess  and  track  the  environment  (weather /terrain, 
social / economic / political  situation) 

b)  Project  environment 

c)  Estimate  and  track  threat  location  and  identity 

d)  Estimate  and  track  adversary  force,  intelligence,  surveillance  and 
reconnaissance  (ISR),  logistics,  communications,  facility  status, 
organisation,  activity  and  beliefs 

e)  Estimate  adversary  objectives,  vulnerabilities.  Courses  Of  Action  (CO As), 
and  plans 

f)  Project  the  threat,  including  possible  COAs 

g)  Estimate  own  force  location  and  capabilities,  the  status  of  own  ISR, 
logistics,  communications  and  facilities 

h)  Estimate  own  force  vulnerabilities  and  opportunities 

i)  Project  own  forces,  including  strengths  and  vulnerabilities 

S-4.  Manage,  integrate  and  distribute  the  situation  picture 

a)  Provide  information  assurance 

b)  Manage  uncertainty 

c)  Assemble  and  integrate  information  for  situation  picture 

d)  Manage  inconsistencies  among  geographically  distributed  pictures 

e)  Tailor  picture  to  support  decision  making 

f)  Share  the  picture 
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Table  2  cant'd. 


Planning  Functions 

P-1.  Prepare  for  planning 

a)  Understand  mission  and  tasks 

b)  Form  planning  team 

P-2.  Develop  strategy 

a)  Understand  objective  trade-offs 

b)  Understand  the  situation 

c)  Develop  concepts  for  achieving  goals 

d)  Identify  goal,  assets,  constraints,  phases,  and  contingencies 

e)  Select  COA 

P-3.  Make  /  revise  tentative  plan 

a)  Decompose  strategy  to  task  and  ID  success  conditions 

b)  Identify  needed  assets:  equipment,  information,  consumables,  and 
organisations 

c)  Determine  boundaries  (geographic,  ROE) 

d)  Schedule  tasks  and  assets  (time,  order) 

e)  Plan  for  automated  and  contingent  decisions 

f)  Plan  for  execution  monitoring  and  success  forecasting 

g)  Evaluate  plan:  suitability,  feasibility,  acceptability,  and  robustness 

P-4.  Deconflict,  finalise,  and  promulgate 

a)  Review  and  coordinate  with  operational  echelons  and  resource 
managers 

b)  Articulate  intent 

c)  Develop  directive 

d)  Promulgate  directive/ share  plan 


4.2  Potential  benefits  of  automated  decision  making  tools 

Automated  decision  making  tools  may  also  be  labelled  as  'automated  SA  tools'  and 
'battlespace  visualisation  tools'  or  'visualisation  tools'  in  various  contexts.  In  essence,  their 
function  and  design  is  intended  to  enhance  the  processes  military  staff  engage  in  when 
making  decisions  regarding  flight  plans,  ingress  or  egress  routes,  potential  threats  from 
enemy  forces  and  a  host  of  other  critical  issues.  The  types  of  visualisation  tools  currently 
being  developed  by  many  researchers  world-wide  may  vary,  however  the  tools  referred  to 
in  the  remainder  of  this  review  will  be  visualisation  tools  intended  to  assist  commanders 
and  their  staff  in  generating  a  dynamic  and  comprehensive  image  of  the  battlefield  and  the 
battle  itself. 
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When  considering  the  potential  uses  for  visualisation  tools,  several  factors  must  be  taken 
into  account,  which  can  then  expand  into  related  areas  such  as  team  traming: 

•  Is  the  tool  relatively  easy  to  learn  to  use? 

•  Is  it  relatively  simple  to  recall  the  instructions  for  use  under  conditions  of  high  time 
pressure /stress,  and  low  training  levels? 

•  Does  the  tool  provide  all  the  functions  previously  carried  out  manually  —  and  if  not, 
why  not? 

•  Does  the  tool  provide  additional  useful  functions,  which  are  difficult  or  impossible  to 
carry  out  manually? 

•  Does  the  tool  facilitate  information/ knowledge  sharing  between  team  members  and 
individual  teams? 

•  Does  the  tool  improve  the  efficiency  of  planning? 

•  Does  the  tool  interfere  with  or  aid  the  uptake,  processing  and  dissemination  of 
situation  relevant  information /knowledge? 

A  major  problem  with  visualisation  tools  -  as  identified  by  Mills  (2000)  -  has  been  their 
tendency  to  interfere  with  team  performance  during  the  initial  introductory  stages.  This 
can  often  be  the  result  of  attempts  to  integrate  somewhat  complex  computer  technology 
into  situations  where  the  users  have  varying  degrees  of  experience  and  competency  with 
such  devices.  In  this  way,  there  can  be  disruption  in  smooth  team  functioning  due  to  the 
shift  in  emphasis  from  the  familiar  manual  methods  to  the  somewhat  unfamiliar  digitised 
methods.  This  disruption  can  -  it  is  predicted  -  be  overcome  with  training  and  increased 
system  and  tool  familiarity,  and  development  of  such  tools  is  ongoing  in  order  to  provide 
a  product  to  the  military  that  fulfils  their  requirements  and  can  be  used  with  a  minimum 
of  frustration  and  stress. 

Currently,  different  iterations  of  the  various  visualisation  tool  solutions  tend  to  include 
functionalities  which  have  been  flagged  as  potentially  important  (or  at  least  desirable)  by 
software  developers.  There  has  also  been  input  from  some  members  of  the  military 
establishment,  and  these  will  be  taken  into  account  in  the  development  process.  Such 
current  developments  include  capabilities  such  as  3-D  fly  throughs,  representation  of 
potential  ingress  routes,  and  the  display  of  threat  domes.  Line  of  sight  tools  are  also 
becoming  common  to  many  of  these  visualisation  tools.  Future  developments  should 
include  progress  towards  ease  of  use,  while  still  incorporating  the  more  comprehensive 
aspects  of  this  technology.  It  seems  that  adding  complexity  to  the  mission  planning  (or  any 
other)  function  by  adding  complicated  software  tools  withnon-intuitive  means  of  control 
may  hamper  the  very  fimctions  that  researchers  are  currently  attempting  to  enhance. 
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5.  The  impact  of  technology  on  planning 

The  impact  of  technology  (eg.  planning  tools  and  automated  decision  aids)  in  terms  of 
distributed  teams  -  i.e.  where  several  members  are  situated  in  separate  physical  locations  - 
should  be  considered  in  light  of  the  combined  operations  of  ground  and  air  teams.  These 
land-air  systems  (such  as  ground  units  combined  with  Armed  Reconnaissance  Helicopters 
[ARH])  need  both  effective  mission  planning  and  communication  of  these  plans  to  the 
relevant  operators.  That  is,  for  such  teams  to  function  as  normal  teams  would,  the 
updating  of  plans  and  tasking  of  units /squadrons  must  be  as  efficient  as  they  would  be  for 
teams  located  at  a  single  site  (Gregory  and  Kelly,  1998). 

Observations  of  both  air  and  ground  mission  planiung  suggest  that  these  two  processes 
involve  somewhat  different  emphases.  That  is,  while  groimd  teams  tend  to  favour 
extensive  pre-task  discussions  and  mission  planning  to  reduce  the  amount  of  information 
exchange  needed  during  the  mission,  air  teams  tend  to  focus  more  on  the  effective  use  of 
communication  during  actual  engagement  with  enemy  aircraft  (Cook,  Angus  and 
Campbell,  1999).  This  is  due  to  the  need  for  aircrews  to  identify,  evaluate  and  execute 
plans  while  in  the  air,  which  leads  to  a  different  operational  approach  to  information 
sharing.  Thus,  mission  planning  for  distributed  teams  (encompassing  both  ground  and  air 
defence)  must  take  into  account  these  different  emphases  and  ensure  that  the  needs  of 
both  groimd  and  air  teams  are  met.  It  is  possible  that  the  introduction  of  automated  tools 
for  increasing  the  efficiency  of  planning  by  groimd  command  teams  can  enhance  the 
operations  of  combined/ distributed  teams  by  allowing  the  faster  processing  and 
representation  of  incoming  information,  as  well  as  the  creation/ modification  of  mission 
plans  and  tasking  for  air  defence  and  reconnaissance  craft  (eg.  the  ARH  squadron).  In  this 
way,  the  capacity  will  then  exist  to  draw  map  overlays  relatively  quickly  and  then  swiftly 
pass  them  to  the  necessary  recipients  (cf.  the  manual  draw  and  copy  method). 

According  to  Cook,  Angus  and  Campbell  (1999:  239)  "the  merging  of  air  defence  and 
ground  attack  roles  has  allowed  users  and  developers  to  accept  that  components  of  each 
mode  of  operation  can  be  adopted  with  some  attrition  of  tasks  which  were  perceived  as 
redundant  in  light  of  new  technologies".  It  should  then  be  possible  for  the  introduction  of 
visualisation  tools  to  provide  avenues  for  enhancing  the  way  that  incoming  and  processed 
information  are  represented  to  the  members  of  planning  teams.  In  addition  they  can  result 
in  an  enhanced  picture  of  the  situation  for  the  commander,  which  may  lead  to  enhanced 
team  SA.  Thus  the  implementation  of  automation  in  a  planning  context  can  potentially  be 
highly  effective  in  improving  the  team  function  and  the  outcomes  of  the  planning  process 
in  terms  of  both  the  time  to  plan  and  the  actual  plans  themselves. 

5.1  Automated  visualisation  tools:  the  potential  benefits 

The  new  visualisations  made  possible  by  the  introduction  of  technological  advances  into 
the  planning  process  have  the  potential  to  make  the  design  and  export  of  mission  plans  to 
other  operators  in  distributed  environments  more  efficient  than  ever  before.  Some  of  the 
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technological  advantages  provided  by  the  new  tools  include  3D  fly-throughs  of 
prospective  mission  plans.  These  can  be  advantageous  for  pilots  and  grovmd  crews  in 
providing  information  on  the  needs  relevant  to  the  mission  being  flown.  The  necessary 
information  for  preferred  flight  paths  could  include  such  things  as  possible  enemy  sight 
lines,  safe  or  relatively  well  hidden  movement  routes,  the  distances  to  be  covered,  and  the 
resupply  necessary  for  weapons,  for  example.  Figure  2  shows  two  possible  line  of  sight 
(LOS)  displays  from  the  perspective  of  either  a  helicopter  (probably  flying  'low  and  slow') 
or  an  elevated  feature  (such  as  a  mountaintop).  The  lines  of  clear  vision  and  those  areas 
hidden  from  view  are  easily  discernible  in  these  figures,  and  would  form  vital  knowledge 
for  the  planning  of  optimal  flight  paths  according  to  the  last  known  intelligence  reports 
available  to  the  pilots/ planners  regarding  enemy  locations. 


V—  - 


Figure  2  a  and  b:  Example  of  LOS  tool  displaying  visibility from  (a)  an  elevated  distant  perspective 
and  (b)from  a  close  and  elevated  perspective  (From  Kirby  et  al  (2000)  Figures  7b  and  7c 
respectively) 

The  increase  in  the  amount  of  information  able  to  be  displayed  in  a  functional  manner  may 
be  perceived  as  a  benefit  -  as  long  as  the  added  information  does  not  create  clutter  or 
mental  work-overload  for  the  tool  operators  (i.e.  the  planners  and  pilots).  A  study  by 
Smallman,  Schiller  and  Mitchell  (1999),  for  example,  foimd  that  when  using  2-D  and  3-D 
display  formats,  the  3-D  format  resulted  in  enhanced  SA  for  the  attributes  of  altitude  and 
attitude.  Thus,  this  would  indicate  a  nett  advantage  for  planners  and  pilots  in  using  a  tool 
with  3-D  capability.  It  was  observed,  however,  that  this  improvement  was  not  immediate. 
Thus,  a  time  lag  in  noticeable  positive  effects  should  be  taken  into  account  when 
evaluating  the  impact  of  any  automation  on  performance  and  SA.  The  benefits  of  3-D 
visualisation  on  warfighter's  understanding  of  the  battlespace  have  been  supported  by  a 
cognitive  task  analysis  carried  out  by  Eddy,  Kribs  and  Cowen  (1999).  This  task  analysis 
indicated,  "...  a  3-D  display  could  provide  a  succinct,  comprehensible  and  readily 
discernible  presentation  of  the  common  tactical  picture."  (p.  iii).  Such  3-D  representations 
can  be  seen  in  Figure  3. 

It  can  also  be  imderstood  that  the  introduction  of  visualisation  tools  can  impact  the  SA  of 
the  teams  (or  teams  of  teams)  endowed  with  the  tools.  The  potential  effect  of  full  or  partial 
automation  is  discussed  in  the  next  section. 
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Figure  3  a  and  b:  Possible  3D  battlespace  visualisations  (From  Ydrby  et  al,  2000:  figures  1  and  2) 


6.  The  impact  of  automation  on  SA 

While  in  many  cases  automation  has  provided  the  desired  benefits  and  extended  the 
system  fimctionality  beyond  the  reach  of  human  capability,  there  has  been  a  change  in  the 
role  of  the  human  operator  (Endsley,  1998)  that  needs  to  be  addressed.  When  humans 
become  relegated  to  monitoring  functions  (that  is,  simply  ensuring  that  the  automated 
tools/systems  are  performing  adequately),  they  tend  to  lose  sight  of  both  the  incoming 
information  and  the  processes  used  to  turn  that  information  into  knowledge.  One  of  the 
benefits  of  the  current  manual  system  of  planning  and  information  processing  is  that  the 
human  stays  'in  the  loop',  and  maintains  up  to  date  awareness  of  what  is  going  on.  It  is 
important  in  the  current  climate  of  piecemeal  automation  (i.e.  partial  automation  of  a 
larger  process)  that  humans — in  their  role  as  integrators — can  maintain  awareness  of  the 
processes  occurring  at  all  stages  of  operations,  and  the  state  of  affairs  outside  of  their  own 
small  area  of  immediate  influence. 

Endsley  (1998)  states  that  automation  impacts  SA  through  three  major  mechanisms: 
changes  in  vigilance  and  complacency  associated  with  monitoring,  assumption  of  a 
passive  rather  than  active  roles  in  system  control,  and  changes  in  the  form  or  quality  of 
feedback  provided  to  the  operator  (Endsley  and  Kiris,  1995).  Additionally,  automated 
systems  (due  to  their  complexity)  challenge  the  higher  levels  of  SA  (specifically 
comprehension  and  projection)  during  ongoing  system  operations  (Endsley,  1998).  Much 
of  this  work  and  the  references  herein  are  to  complex,  dynamic  automated  systems  (eg. 
nuclear  power  plants,  advanced  product  manufacturers,  air  traffic  control  and  medical 
systems).  Planning  teams  within  the  Army  context  will  have  specific  needs  of  their  own 
that  must  be  catered  for  in  order  to  ensure  the  effective  implementation  of  automated 
tools/systems  within  the  planning  loop.  Thus,  there  is  a  need  for  people  using  automated 
decision  aids  in  this  context  to  be  good  information  processors.  There  is  an  indication  then 
that  partial  automation  -  particularly  in  the  case  of  Army  planning  and  decision  making  - 
is  the  desired  option.  That  is,  ultimately,  the  creation  of  mission  plans  and  the  making  of 
critical  decisions  are  better  left  to  the  human  experts  (eg.  the  commanders  overseeing  the 
processes)  than  the  automated  tools  (see  Mosier  and  Skitka,  1998  for  a  review  of  humans 
and  automation). 
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7.  Measures  for  assessing  technological  impact 

The  effects  of  automated  decision  aids  on  the  performance  of  the  planning  team  may  be 
determined  in  several  ways.  In  terms  of  hard  and  soft  team  performance  measures  in 
certain  team  skill  areas,  possible  metrics  (adapted  from  Table  3-1  in  Gregory  and  Kelly, 
1998)  might  include: 

•  Making  maximum  use  of  time  available  for  planning 

•  Providing  clear  and  concise  mission  plans 

•  Planning  for  and  maintaining  flexibility 

•  Synchronising  all  assets 

In  these  areas,  'hard'  measures  are  the  objective  performance  measures  such  as  actual  time 
use  comparisons,  while  'soft'  measures  include  subjective  performance  assessment 
measures  by  trainers  or  observers. 

Other  potential  groups  of  'hard'  measures  include  the  performance  of  the  planning  team 
using  automated  tools.  These  results  can  be  compared  with  prior  performance  without 
automated  tools  in  order  to  gauge  the  tools'  effects.  These  measures  should  be  taken 
longitudinally  in  order  that: 

•  Potentially  useful  tools  are  not  dismissed  out  of  hand  due  to  poor  initial 
performance. 

•  The  learning  curve  of  the  planning  team  can  be  tracked. 

•  This  learning  curve  can  be  applied  to  predict  future  teams'  training  time  scales 
with  the  same  tool. 

Some  of  the  performance  measures  may  include  items  such  as  the  time  taken  to  perform 
specific  actions,  the  number  of  movements  required  to  perform  actions^,  and  the  t)q)es  and 
numbers  of  errors  being  committed  by  human  operators  of  the  tools.  Measures  of  the 
operator's  workload  both  before  and  after  tool  introduction  can  also  be  taken,  as  well  as 
attention  load  on  individual  operators.  It  is  the  author's  intention  to  combine  a  variety  of 
these  measurements  to  form  a  set  of  paper  'automation  evaluation'  tools  applicable  to  a 
variety  of  situations.  That  is,  situations  in  which  it  is  desirable  to  gauge  the  effect  (or 
potential  effect,  when  dealing  with  concept  demonstrators)  of  new  tools  on  team 
performance.  As  there  may  be  an  interaction  between  the  decision-making  styles  of  novice 
and  expert  teams  and  commanders,  it  is  also  critical  that  the  methods  used  be  understood 
and  evaluated  for  their  outcomes  prior  to  any  introduction  of  automation.  This  way,  there 
can  be  a  truer  evaluation  of  the  contribution  of  the  tools  to  the  outcomes  and  processes. 


5  These  both  relate  to  Human  Machine  Interface  issues  that  can  also  be  examined  by  interface 
design  evaluations.  Any  problems  encountered  can  be  addressed  in  recommendations  to 
manufacturers  of  automated  tools. 
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7.1  Qualitative  and  quantitative  assessment  of  SA  and  planning 
outputs 

There  are  several  potential  methods  for  measuring  the  critical  points  in  the  processing  and 
handling  of  information,  and  the  application  of  the  developed  knowledge  where  Human 
Factors  evaluations  can  be  conducted  on  the  impact  of  automation  on  SA  and  mission 
planning.  Potential  measurement  methods  here  include: 

•  A  modified  TARGETS^  methodology  (MTM):  (refer  to  Dwyer,  Oser,  Salas  and 
Fowlkes,  1999;  Fowlkes,  Lane,  Salas,  Franz  and  Oser,  1994;  Prince  and  Salas,  1999) 

•  The  TOM  instrument  (Team  Observation  Measure):  (refer  to  Dwyer,  Oser,  Salas 
and  Fowlkes,  1999;  Fowlkes,  Lane,  Salas,  Franz  and  Oser,  1994) 

•  Questionnaires  to  SMEs 

•  Behavioural  observation 

7.1.1  The  Modified  TARGETS  Methodology 

The  TARGETS  methodology  appears  to  be  a  good  and  fairly  objective  way  of  measuring 
team  performance;  there  is,  however,  an  issue  of  concern  with  the  implementation  of  this 
methodology  in  an  operational  environment  or  during  large-scale  military  exercises.  These 
two  settings  are  vital  for  the  gathering  of  data  under  high  fidelity  conditions,  so  that  the 
findings  can  better  be  generalised  to  actual  military  operations;  however  inserting  events 
into  an  actual  operational  situation  is  nigh  on  impossible,  and  into  a  large-scale  military 
exercise  is  still  very  difficult.  Therefore,  a  modified  TARGETS  methodology  is  proposed 
for  such  situations  with  the  intention  of  providing  a  way  of  accessing  TARGETs-type 
information  in  a  manner  that  is  appropriate  for  use  in  the  operational  military 
environment. 

This  modified  method  involves  combining  a  reordered  version  of  the  original  TARGETS 
methodology  with  observational  techniques  to  produce  critical  sample  events  (or  a  series 
of  these)  from  a  scenario  or  series  of  military  operations.  These  are  then  examined  by 
SMEs  and  a  list  of  acceptable  responses  (behavioural  or  otherwise)  for  these  sample  events 
is  produced.  The  list  can  then  be  compared  with  the  responses  that  actually  occurred  at  the 
time  of  occurrence  of  each  sample  event/ series  to  determine  the  level  of  agreement  with 
the  generated  list.  This  can  also  be  compared  with  a  list  of  responses  from  the  military 
participants  in  the  event  to  provide  a  list  of  other  acceptable  responses  (i.e.  other  than  the 
response  that  has  already  occurred). 


6  The  TARGETS  methodology  refers  to  the  Targeted  Acceptable  Responses  to  Generated  Events 
tool,  which  is  a  checklist  that  can  be  individually  tailored  to  each  scenario  by  researchers  in 
conjunction  with  operational  experts.  It  consists  of  a  list  of  expected  behaviours  or  responses  from 
crew  or  team  members  in  response  to  events  within  the  scenario.  The  occurrence  (or  otherwise)  of 
the  expected  behaviour  is  checked  off  during  the  course  of  the  scenario  by  raters  with  no 
judgements  made  about  the  behaviour.  This  is  simple  to  use  but  a  labour  intensive  tool  to  construct. 
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7.1.2  The  Teamwork  Observation  Measure 

The  Teamwork  Observation  Measure  (TOM)  is  derived  from  several  performance 
measurement  techniques  developed  during  such  research  as  the  Navy's  TADMUS  project 
(Johnston,  Smith-Jentsch  and  Cannon-Bowers,  1997)  and  aircrew  coordination  and  training 
(Prince  and  Salas,  1993).  This  instrument  was  designed  to  test  performances  for  strengths 
and  weaknesses,  and  to  rate  performances  on  critical  teamwork  dimensions  (Dwyer,  Oser 
and  Fowlkes,  1995).  From  empirical  validations  of  this  methodology,  four  dimensions  of 
teamwork  have  been  identified:  communication,  team  coordination,  SA,  and  team 
adaptability.  Each  of  these  dimensions  was  divided  into  separate  categories,  and  assessors 
instructed  to  provide  comments  during  their  observations  to  be  used  for  critical  teaching 
points  during  feedback.  These  included  such  items  as  'out  of  sequence  communications'. 
The  TOM  also  includes  an  assessor  rating  scale  for  interactions  within  the  team  on  the  four 
dimensions  of  teamwork  (Dwyer,  Oser,  Salas  and  Fowlkes,  1999). 

7.1.3  Questionnaires  to  SMEs 

SMEs  can  be  a  source  of  critical  information  for  both  the  progress  of  developing  systems, 
and  the  evaluations  performed  on  those  systems.  Thus  the  knowledge  SMEs  hold  should 
be  tapped  early  into  the  evaluation  cycle,  particularly  if  field  data  is  difficult  to  obtain  or 
the  system  is  still  at  the  schematic  stage. 

7.1.4  Behavioural  Observation 

This  is  a  method  commonly  used  by  behaviourists  world  wide,  and  can  be  extremely 
useful  in  the  military  context.  It  is  essential  for  providing  both  a  snapshot  of  the 
behaviours  occurring  in  the  situation(s)  under  investigation  and  a  taxonomy  of  the 
behaviours  involved  in  given  situations,  and  can  include  the  timing  of  given  behaviour, 
the  frequency  of  occurrences  of  certain  behaviours,  and  the  sequences  of  behaviours.  The 
appropriate  data  can  then  be  decided  upon  by  the  researcher(s)  according  to  the  purpose 
of  the  evaluation.  In  examining  the  impact  of  automation  on  mission  planning,  for 
example,  researchers  may  take  note  of  the  time  taken  to  use  system  components,  the  order 
in  which  planning  behaviours  are  performed,  and  the  frequency  of  vital  planning 
behaviours.  These  things  may  then  be  compared  in  planning  teams  with  and  without 
automated  tools,  or  indeed  those  with  outdated  tools  compared  with  demonstrations  of 
future /developmental  tools.  The  focus  is  both  at  the  individual  and  at  the  team  level, 
where  team  level  issues  can  be  identified  from  the  timings  of  behaviour,  the  levels  of 
behaviours  overall,  and  the  outcomes  of  these  conditions. 
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8.  Conclusions 


The  understanding,  measurement,  cmd  facilitation  of  SA  development  and  maintenance  in 
military  teams  (and  teams  of  military  teams)  is  not  a  simple  matter.  In  addition,  it  is  no 
longer  simply  a  case  of  examining  the  team  members  themselves  -  there  are  other  agents 
to  be  considered  in  the  modem  Army.  Thus,  where  the  study  of  shared  SA  is  a  complex 
issue  when  teams  of  humans  are  involved,  the  interaction  of  such  teams  with  introduced 
automation  presents  an  ever  more  complex  picture  with  which  researchers  are  struggling 
to  come  to  terms.  The  ultimate  aim  of  such  technological  advance  is  improvement  of  team 
performance  (through  enhanced  SA)  and  the  aiding  of  military  functions  such  that 
friendly  forces  have  the  edge  when  it  comes  to  confrontation  with  enemy  forces.  In  order 
to  ensure  that  performance  enhancement  is  what  is  actually  occurring,  research  needs  to 
delve  into  the  reality  of  such  automation  and  its  impact  on  the  performance  of  the  teams 
using  it.  This  will  be  the  subject  of  future  research,  in  addition  to  related  areas  such  as  skill 
development  in  the  use  of  such  tools,  the  potential  training  lead  time  necessary  to 
maximise  human  performance  in  conjunction  with  these  tools,  and  optimising  the  design 
of  computer  interfaces  to  facilitate  ease  of  use. 
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